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Abstract 

This present study focused on examining the levels of questions in Indonesia tertiary education. A survey research was 
conducted in one of the private universities in North Sumatra. The English summative assessment in an undergraduate 
education was used as target of survey. There were a collection of questions that had been administered by four English 
lecturers from Faculty of English Education in Universitas Muslim Nusantara (UMN) A1-Washliyah Medan, Indonesia. 
The sample of research included the sixty five questions of summative tests. The qualitative content analysis which is 
based on Gallagher/Aschner Bloom Classification System used to categorize the questions. It was found that the lower 
order thinking level still dominated the question types (69%). There was only 31% higher order questions used on the 
summative test. 
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1. Introduction 

Questioning students is regularly used by educators as a pedagogical approach and an instrument of assessment. It is an 
indispensable expertise which the teacher or instructors must possess; regardless their style of teaching or the lesson 
content (McNamara, 1981, p. 106). Unluckily, Hill and Flynn (2008) reveal that remembering factual records is what 
many teachers mostly employ when they test the students. This practice focusing in lower order level questions might 
not contribute to the development of students’ cognitive skill in reasoning and analyzing (pp. 48 - 49). Bloom 
taxonomy (as cited in Forehand, 2010) established a scheme for sorting questions into different thinking domains. The 
six classified stages consist of knowledge, comprehension, application, analysis, synthesis, and evaluation. It is graded 
from the least to the most difficult logical processes. 

Wilen (1991) also develops the Gallagher-Aschner/Bloom hybrid system which will be employed in this study to 
categorize the questions level. This scheme is a simplified and has been appropriately used to examine teachers’ 
questioning level. Unlike, the six hierarchical questions in the original Bloom taxonomy; there are only four levels of 
questions in this classification. The questions are mainly assigned into convergent and divergent questions. The 
convergent questions’ purpose is to assess students’ basic knowledge and understandings (lower order). On the other 
hand, divergent questions oblige students to employ in higher-order thinking process as seen in Table 1. 


Table 1. Teachers’ Questioning Levels by Willen (1991) 



Levels 

Definitions 

Purposes 


Examples 

Level 

I - Low 

This is equal with 

The teacher's major purpose is to 

1. 

Define the term 

Order 

Convergent: 

Knowledge level in 

demand student’s ability to 





Bloom’s taxonomy 

remember or memorize answers 

2. 

What is a ? 



(McComas & Abraham, 

which has already been definitely 

3. 

Who did ? 



2004). 

delivered in classroom. 


Name 

Level 

II - High 

Comprehension and 

Learners are asked to display 

1. 

How will you interpret in 

Order 

Convergent 

Application levels in 

further than recall skill but ability to 


your own words... ? 



Bloom’s taxonomy are 

apply the information and exhibit 

2. 

What is the main idea of 



measured to be in this 

understanding 


? 



level 




Level 

III -Low 

This is equal to Analysis 

The teacher's purpose is to require 

1. 

What is the relationship 

Order 

Divergent 

level in Bloom’s 

learners to analyze the grounds or 


between . . . ? 



taxonomy 

reasons, draw suppositions or to 

2. 

What are some possible 




support an argument. 


consequences? 

Level 

IV - High 

Synthesis and Evaluation 

Higher-order questions demanding 

1. 

Why did they (the 

Order 

Divergent 

of Bloom Taxonomy are 

students to come up with solutions 


character) choose.. .? 



graded in this level 

for substantial problems. Produce 

2. 

Create a poster to 




innovative ideas and practical 


promote a.... 




actions. 
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Some studies on this field have been done, for examples; Soleimani and Khairi (2016) examined Iranian testing classes 
quality by using recorded corpus acquired from MA testing classes and found that ten university professor in MA class 
used lower order thinking questions (69, 445%), medium (30, 55%) and no higher order questions thinking at all (pp. 
867 - 868). Inamullah (2011) conducted a research in Pakistan secondary school and found that most of the questions 
asked are low order thinking from the total of 267 questions, 67 percent were dominated by knowledge based questions 
(p. 151). A case study in a Chinese university carried out by Shen and Yodkhumlue (2012) also showed that during their 
classroom observation, teacher asked 79, 2% questions and only 20, 8% higher order questions (p.201). Shuhidan, 
Hamilton and D’Souza (2009) observed questions of summative assessment in the programming subject by applying 
Bloom Taxonomy and found out that the majority questions are in the comprehension level (p. 150). In contrast, by 
using qualitative approach and test - driven content analysis, Tikkanen and Aksela (2012) examined cognitive skills 
questions of Bloom Taxonomy in Finnish Chemistry examination and found that the questions are immensely 
dominated by higher order cognitive skills (p.263). Okanlawon and Adeoti (2014) also used qualitative approach and a 
content analysis to investigate cognitive complexity of West African Senior School Certificate Chemistry Examination 
questions and recommended that lower order and higher order questions must be distributed evenly in the test (p.19). 

However, most of the studies above had been conducted in the field of non - English lessons. The studies pertaining to 
the level of English summative questions made by teachers or lecturers that rarely documented. It might be related to 
the belief that English learners still have inability in the deep thinking (Harklau, 2000). On the other hand, 
communicating in English requires speakers to respond and express ideas in a natural way instead of remembering 
written expressions or structures. Yet, many English teachers still assume that their ESL students cannot answer the 
questions if it is made in higher order (Anil, 2015; pp. 47-48). Thus, this belief will affect the choice of question 
employed by teachers in their tests. There are some reasons why it is so pivotal for language teachers to develop the 
higher order thinking of questions that enforce their students do higher their analysis. It is because literate student is not 
only those who can read language but also those who can think critically (Nagappan, 2001).Thus, teaching skills in 
language classroom should also about developing students’ thinking skills. Redfield and Rousseau (1981) argue that 
questioning students can be used to enhance critical thinking. Language achievement in skill lesson will also be 
progressively increased as thinking competencies are strengthened (Collin, 1991). DeWaelsche (2015) reveals that some 
factors like English proficiency limitation, socio - cultural and institutional reasons can affect Korean university 
students’ inability to participate in higher order thinking skills. These can also be some fundamental concerns for 
English language instructor in Korea to formulate their questions (pp. 144 -145). Those reasons might be similarly 
found in some other Asian countries. Vogler (2005) also states that when it comes to verbal questioning, teachers still 
face problem due to having lack of knowledge and understanding of the importance of higher cognitive questions (p. 
98). It is not only faced by the novice teachers; because, apparently, experienced teachers also need the skill of 
formulating questions more than just testing factual knowledge (Pagliaro, 2011). The lack of teacher training programs 
which is intended to support teachers with the knowledge of questioning skills might be one of the problems (Wilen, 
1991) 

Only a few related researches on the infusion of higher order thinking questions, particularly in Indonesian tertiary 
education, analyze the language tests. Some studies are commonly conducted in lower level education. For example, a 
survey research conducted by Iskandar & Senam (2015) found that 15 chemistry teachers in senior high schools infused 
only 13, 9% higher order thinking questions in their summative assessment. Sunggingwati & Nguyen (2013) also report 
that the Indonesian secondary school teachers’ who tend to employ more low order thinking questions in reading 
lessons and need support to face challenges in creating higher order questions (p.89). The World Bank published a book 
called “ Skills for the Labor Market in Indonesia: Trends in Demand, Gaps, and Supply” by Di Gropello, Kruse and 
Tandon (2011) and report that Indonesian secondary school graduates have gaps in English, thinking and other practical 
skills. It also describes that Indonesian youth lack of skill preparedness which turns into poor outcome in employment. 
Thus, one of their recommendations is to develop quality and relevance of higher education. One interesting statement 
is ‘‘although of generally better quality, the skill of higher-education graduates also need improvement, with particular 
emphasis on higher order thinking skills, English, and job specific skills (p-148) 

Therefore, this present study is aimed to explore the gap by investigating the level of questions used by English 
Education lecturers in their summative assessment. The decision to choose summative assessment is because it has a 
pivotal role for it is executed to measure and record students’ attainment towards the end of an instructional component 
(McMillan, 2008; Nitko& Brookhart, 2011). Therefore, it is imperative to see what kinds of outcome lecturers plan to 
have on their English skills subject. The choices of questions in their summative tests surely reflect the teaching goals 
that the lecturers have at the end of the term, particularly on what the type of cognitive skills they target to assess. This 
could also be an avenue to look at the lecturers questioning preferences which eventually lead to the evaluation because 
summative assessment is known to benefit educators particularly in determining how their instructional and learning 
objectives have been successfully met (Johnson & Jenkins, 2009). Thus, the research question of this research paper is: 
“Which level of question that mostly found in Indonesian’s English Language Teaching (ELT) ?” 

2. Research Methodology 

This research was applied a survey technique in Universitas Muslim Nusantara (UMN) A1 Washliyah Medan, 
Indonesia. A qualitative content analysis was utilized to categorize the levels of questions of ELT tests. Mayring (2002) 
argues that the qualitative content analysis objects can be any type of recorded communication (audio and video tapes, 
interviews transcripts, discourses, and documents). The data of this research were the documents from the collection of 
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question - sheets of summative tests of 2015 - 2016 and 2016 - 2017 academic year of four English lecturers from the 
Faculty of Education UMN Al-Washliyah Medan. There were four lecturers chosen through three criteria, namely: 1) 
they have been teaching in the university at least for two years and half (it is important as this duration of teaching has 
supposedly given them adequate time for doing at least four successive times of summative test) 2) The subjects they 
teach are English but not in the area of grammar, pronunciation or vocabulary 3) Instead, four of them are teaching skill 
- lessons such as speaking, writing and reading. The reason for excluding grammar, pronunciation or vocabulary lesson 
is dues to the higher order thinking questions. The skill lessons might be assumed to generate more output in higher 
order thinking questions. 

In order to collect the data above, the researchers made the process of collecting the data in two semesters. Then, as 
long as that time, there were 65 questions from four English lecturers were gathered, and all the data transcribed into 
teacher’s questioning levels. Furthermore, the data found would be analyzed by using Gallagher/Aschner Bloom 
Classification System. Each question would be coded into four different level of thinking skills and analyzed to find out 
level of question is mostly employed by English lecturers in their summative assessment. Thus, the involvement of 
another analysis was employed to do a counter - check of the content data analysis to increase the reliability of the 
study. 

3. Finding And Discussion 

3.1 Finding 

An analysis of the question sheets data pointed out that most of the questions asked by the lecturers were lower-order 
questions. There are total 65 questions observed which were taken from English reading, writing and speaking related 
subjects summative test sheets. Data analysis showed that only 31% of the whole questions were directed to Level 3 
(Low Order Convergent) and Level 4 (High Order Convergent) questions as shown in the Table 2. 


Table 2. Levels of Questions asked by the participant 

in the four observed lessons 


Levels of Higher Order Questions 

Total 

Percentage (%) 

Level 3 

5 

5% 

Low Order Divergent 

Level 4 

15 

26% 

Higher Order Divergent 

Total 

20 

31% 


The table above obviously shows that only 31% of higher order questions included. There were 5% Level 3 lower order 
divergent and 26% level 4 higher order divergent. Theoritically, level 3 question refers to analysis in which the lecturers 
require the students to analyze, to present reasoning, and to support an argument. While, level 4 is the highest level 
points out synthesis and evaluation. In this highest level, the lecturers demand the students to be able to produce the 
innovative ideas and to offer solution and practical actions towards a substantive problem. In brief, the finding slightly 
shows that some lecturers tend to require the students to think critically. In contrast, lower order question is visually 
shown in Table 3. 


Table 3. Levels of Lower-order Questions asked by the participant in the three observed lessons 


Levels of Lower-order Questions 

Total 

Percentage (%) 

Level 1 

31 

55% 

Low Order Convergent 

Level 2 

14 

14% 

Low Order Convergent 

Total 

45 

69% 


The result of data analysis above shows that most lecturers dominantly used lower-order questions. There was 55% of 
level 1 lower order question. As Gallagher/Aschner Bloom Classification System stated, level 1 indicates that the 
questions demanded students’ basic knowledge and understanding in which the lecturers tested the students solely based 
on the ability to memorize a particular idea. In addition, there was 14% of level 2 question found in this research which 
means that the lecturers have requested the students to apply some formula or pattern of structures and to exhibit 
understanding. In short, the comparison between high-order questions and low-order questions is described in details in 
Table 4. 
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Table 4. The Comparison of Levels of Questions 


Levels of Lower-order Questions 

Total 

Percentage (%) 

Lower order Questions 

45 

69% 

Higher order Questions 

20 

31% 

Total 

65 

100% 


The results indicated that there was a vast difference between the number of lower-order questions and higher-order 
questions enquired. The outcomes in the present study are consistent with those of existing literatures (Hill and Flynn, 
2008; Inamullah 2011;Soleimani and Khairi 2016). As lower-order questions ware dominantly found in this research, 
there are two possible underlying factors. Firstly, the lecturers of the Faculty of English Education UMN Al-Washliyah 
Medan Indonesia might have lack of understanding on the importance of infusing higher order questions into their 
summative assessment. Secondly, the lecturers probably have low perception of their own students. They possibly 
doubt whether students can answer the higher order questions or not. Surely, these factors will significantly affect 
students’ critical thinking in the long term. Students need to be introduced with higher order thinking questions in order 
to improve their thinking ability particularly in tertiary level of education. If this is a bit overwhelming to create, 
balancing the lower and higher order questions in a test can be an option. Students must not only be given questions 
which only test them to recall information but also to evaluate and to innovate. Critical thingking skill does not only 
allow higher education students to think critically, but it also allows them to solve the problem, to make critical 
arguments and to come up with some cutting - edge ideas. These skills are surely valuable for them to compete in the 
future employment opportunity. 

4. Discussion And Conclusion 

After conducting data analysis, the research finding showed that four levels of higher order thinking question appeared 
in analyzing data, namely; level I- low order covergent, level II-high order covergent, level Ill-low order divergent, and 
level IV-high order divergent. These four levels of question are formed from six classified stages, namely; knowledge, 
comprehension, application, analysis, synthesis and evaluation. In fact, the most dominant questions asked by the 
lecturers were lower-order question from the total of 65 questions analyzed, 55% were dominated by level-I question 
which refers to the knowledge level as shown in data 1. 

Data 1: 

What is the definition of prediction in reading comprehension? 

Every’ body has .. (know) that she’s a bad girl, (past participle form) 

Give an example of fantasy and realistic fiction! 

Furthermore, 14% belongs to level-II which demands the students to comprehend and to apply a particular concept as 
seen in data 2. 

Data 2 : 

Choose a subject as noun phrase and a verb, then match them with an adverb phrase on the right to make 
a good sentence 

Find the cause and effect in this sentence. Underline the signal word! 

Arrange paragraphs about your family at least 150 words! 

Then, there was 5 % of level-III higher order question which focuses on analysis skill as seen in data 3. This level is the 
least type of question chosen by the lecturers. 

Data 3: 

Why do you need to study Writing Skill? Please elaborate clearly! 

How do you explain the relationship between public speaking and ethical responsibility? 

Lastly, there is 26 % of level-IV question which demands the ability of students to synthesis, to evaluate and to create 
innovative ideas as seen as follows. 

Data 4 : 

Please make AREL (argument, reasoning, examples, link-back) from these following motions! 

Your lecturer asks all students to convey some suggestions relating to the topic discussed, yet all students 
seem to be silent including you. This situation makes the lecturer keeps encouraging you all to speak up 
without considering the grammar mistake while you are speaking. What do you think of this situation ? Do 
you have an intention to speak up ? why or why not ? 

This finding of the researh is similar to research conducted by Soleimani and Khairi (2016) which found that ten 
university professors in MA class mostly used lower order thinking question (69,445%). Although, this result does not 
echo, Tikkanen and Aksela (2012) study reveals the questions in Finnish Chemistry examination are dominated by 
higher order cognitive skills. Similar research was conducted by Iskandar & Senam (2015) and it states that there was 
only 13,9% higher order questions made by chemistry teachers of senior high schools in their summative assesment. 
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English is particularly a subject which does not only ask the students to remember structures but most importantly, to 
express ideas. Therefore, treating students with more challenging questions which trigger their thinking skills is more 
encouraged. The lower order questions may be still be given but this has to be less allocated as the semester ascends. 
English major skill (reading, writing, speaking, and listening) tests must not be given in form of remembering 
information because it does not last permanently. When the students recall the answers, they basically don’t create new 
thoughts instead they try to solely tell some exact information which have been stored in the memory. This information 
might have already been provided on the textbooks. This recalling factual information level does not train them to 
produce an idea which is conflicted to the notion of learning a language. Particularly for adult learners who are not 
supposed to be given a model of rote memorization on their language learning. Thus, the choice of choosing the right 
type of question still plays an important role in improving students’ critical thinking especially in English subjects. 

Critical thinking skill will absolutely give a great benefit for those who are studying in undergraduate school since the 
students will have the opportunity to prepare themselves for their future working life. As one of the issues is the lack of 
teacher’s knowledge in questioning skill, therefore, they are suggested to attend workshop and training program in order 
to have an understanding and ability to produce different level of questions which can be instilled through either their 
formative or summative test. However, there must be more emphasis given on summative test as it is a final product of 
a particular teaching period which sums up the learning objectives of a subject. This is a pivotal benchmark for lecturers 
or teachers as it also highlights what elements of thinking ability they have been developing during the semester. Thus, 
if the tests are still dominated with lower order questions then it is hard to yield a decent outcome in terms of graduates’ 
higher order thinking ability. Particularly, English proficiency requires students to expand their ideas critically rather 
than only memorizing scripted structures. This tertiary level of education is a stage for students to prepare themselves 
before getting into the real world of work search or further academic programs. Employing the appropriate questions 
will trigger them to have strong analysis and reasoning ability which have become the key capitals in finding better 
future opportunities in both educational or employment. 


Acknowledgment 

The authors would like to express sincere gratitude to Indonesia Endowment Fund for Education (LPDP) Ministry of 
Finance of the Republic of Indonesia for the financial support. 


References 

Anil, B. (2015). Higher-Order Questioning in SL classrooms-a study. The Buckingham Journal of Language and 
Linguistics, 1,47-55. 

Collins, C. (1991). Reading instruction that increases thinking abilities. Journal of Reading, 34(1), 510-516. 

DeMarrais, K. (2004). Qualitative interview studies: Learning through experience. Foundations for research: Methods 
of inquiry in education and the social sciences, 51-68. 

Di Gropello, E., Kruse, A., & Tandon, P. (2011). Skills for the labor market in Indonesia: trends in demand, gaps, and 
supply. World Bank Publications. 

DeWaelsche, S. A. (2015). Critical thinking, questioning and student engagement in Korean university English 
courses. Linguistics and Education, 32, 131-147. 

Forehand, M. (2010). Bloom’s taxonomy. Emerging perspectives on learning, teaching, and technology, 41-47. 

Harklau, L. (2000). From the “good kids” to the “worst”: Representations of English language learners across 
educational settings. TESOL quarterly, 34(1), 35-67. 

Hill, J. D., & Flynn, K. (2008). Asking the right questions. Journal of Staff Development, 29(1), 46-52. 

Inamullah, H. M. (2011). A study of lower-order and higher-order questions at secondary level. Asian Social 
Science, 7(9), 149. 

Iskandar, D., & Senam, S. (2015). Studi kemampuan guru kimia sma lulusan UNY dalam mengembangkan soal UAS 
berbasis HOTS. Jurnal Inovasi Pendidikan IPA, 7(1), 65-72. 

Johnson, E., & Jenkins, J. (2009). Formative and summative assessment. Psychology of classroom learning: An 
encyclopedia. Detroit: Macmillan Reference. 

Mayring, P. (2000). Qualitative Content Analysis. Forum Qualitative Sozialforschung / Forum: Qualitative Social 
Research, 7(2). Retrieved from http://www.qualitative-research.net/index.php/fqs/article/view/1089/2385 

Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. John Wiley & Sons. 

McComas, W. F., & Abraham, L. (2004). Asking More Effective Questions. Rossier School of Education. 

McMillan, J. H. (2008). Assessment essentials for standards-based education. Corwin Press. 

McNamara, D. R. (1981). Teaching Skill: the Question of Questi. Educational Research, 23(2), 104-109. 

Nagappan, R. (2001). Language teaching and the enhancement of higher-order thinking skills. Anthology series- 
SEAMEO regional language centre, 190-223. 




31 


ALLS 8(3):26-31, 2017 

Nitko, A. J., & Brookhart, S. M. (2011). Educational Assessment of Students. Boston: Pearson Education, Inc. 

Okanlawon, A. E., & Adeoti, Y. F. (2014). Content analysis of West African senior school certificate chemistry 
examination questions according to cognitive complexity. Ife Psychologia, 22(2), 14. 

Pagliaro, M. M. (2011). Exemplary classroom questioning: practices to promote thinking and learning. R&L 
Education. 

Redfield, D. L., & Rousseau, E. W. (1981). A meta-analysis of experimental research on teacher questioning 
behavior. Review of educational research, 51(2), 237-245. 

Shen, P., & Yodkhumlue, B. (2012). A Case Study of Teacher's Questioning and Students' Critical Thinking in College 
EFL Reading Classroom. International Journal of English Linguistics, 2(1), 199. 

Shuhidan, S., Hamilton, M., & D'Souza, D. (2009). A taxonomic study of novice programming summative assessment. 
In Proceedings of the Eleventh Australasian Conference on Computing Education-Volume 95 (pp. 147-156). Australian 
Computer Society, Inc. 

Soleimani, H., & Kheiri, S. (2016). An Evaluation of TEFL Postgraduates' Testing Classroom Activities and 
Assignments Based on Bloom's Revised Taxonomy. Theory and Practice in Language Studies, 6(4), 861. 

Sunggingwati, D., & Nguyen, H. T. M. (2013). Teachers’ questioning in reading lessons: A case study in 
Indonesia. Electronic Journal of Foreign Language Teaching, 10(1), 80-95. 

Tikkanen, G., & Aksela, M. (2012). Analysis of Finnish chemistry matriculation examination questions according to 
cognitive complexity. Nordic Studies in Science Education, 8(3), 257-268. 

Vogler, K. E. (2005). Improve your verbal questioning. The Clearing House: A Journal of Educational Strategies, 
Issues and Ideas, 79(2), 98-103. 

Wilen, W. W. (1991). Questioning skills, for teachers. What research says to the teacher !National Education 
Association. 

Zhang, Y., & Wildemuth, B. M. (2009). Unstructured interviews. In B. Wildemuth (Ed.), Applications of social 
research methods to questions in information and library science (pp.222-231). Westport, CT: Libraries Unlimited. 




